[libtorch] Update to 2.13 - #52815
Conversation
Dependencies.cmake's system-xnnpack fallback (used because USE_SYSTEM_LIBS forces USE_SYSTEM_XNNPACK=ON while USE_XNNPACK itself stays off) declares XNNPACK/microkernels-prod as SHARED IMPORTED, but vcpkg's xnnpack port always builds them statically (vcpkg_check_linkage(ONLY_STATIC_LIBRARY)). CMake only enforces IMPORTED_IMPLIB for SHARED imported targets when an actual link step needs them, so this only failed on Windows dynamic-linkage triplets (x64-windows, x64-windows-release) with a generate-time "IMPORTED_IMPLIB not set" error; static-linkage triplets and non-Windows platforms tolerated the mismatch silently. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PyTorch 2.13 added cmake/PackageData.cmake, included unconditionally
from the top-level CMakeLists.txt. It installs the Python package's
data files using destinations relative to CMAKE_INSTALL_PREFIX, which
is correct for a wheel build (where the prefix maps to torch/) but
under vcpkg resolves to the package root.
The type-stub rule is the worst offender:
install(DIRECTORY "${TORCH_SRC_DIR}/" DESTINATION "."
FILES_MATCHING PATTERN "*.pyi" PATTERN "py.typed")
FILES_MATCHING still creates the whole directory skeleton, so the
install produced 347 empty directories plus py.typed in the package
root, failing both the empty-directories and misplaced-regular-files
post-build checks on every triplet that builds this port.
The module installs nothing but Python package data and the portfile
already configures with -DBUILD_PYTHON=OFF, so guard the include
rather than deleting hundreds of paths after the fact.
Verified on x64-linux: the port builds with no post-build check
problems, the installed tree has no empty directories and no stray
package-root entries, and a consumer linking torch builds and runs.
Two CI regressions in the 2.13.0 update:
1. glog (arm64-osx, x64-windows, x64-windows-release)
pytorch 2.13 added a second use of glog's internal
google::glog_internal_namespace_::IsGoogleLoggingInitialized() in
c10/util/Exception.cpp (WarningHandler::process). fix-glog.patch only
rewrote the existing call in c10/util/Logging.cpp, so c10 failed to link:
Exception.cpp.obj : error LNK2019: unresolved external symbol
"bool __cdecl google::glog_internal_namespace_::IsGoogleLoggingInitialized(void)"
Apply the same treatment to Exception.cpp: drop the hand-rolled
declaration of the internal symbol and call the public
google::IsGoogleLoggingInitialized(), which glog 0.7.1 exports from
<glog/logging.h> as GLOG_EXPORT.
2. CUDA (x64-linux, x64-windows-release)
Configure aborted with:
CMake Error at cmake/FileMirroring.cmake:56 (message):
CuTeDSL source not found at .../third_party/cutlass/examples/python/
CuTeDSL/blackwell/grouped_gemm.py.
FileMirroring.cmake exists purely to populate the Python package tree
(SKBUILD_PLATLIB_DIR), and it hard-errors on USE_CUDA when the bundled
cutlass submodule is absent -- which it always is, since the port uses
system nvidia-cutlass. fix-python-package-data.patch already guarded the
sibling include(cmake/PackageData.cmake) with BUILD_PYTHON; guard
include(cmake/FileMirroring.cmake) the same way.
This also stops the mirroring install() rules from writing into
PROJECT_SOURCE_DIR, which is what SKBUILD_PLATLIB_DIR falls back to when
scikit-build-core is not driving the build.
The port always configures with -DBUILD_PYTHON=OFF (passed after
FEATURE_OPTIONS), so neither mirroring nor package data is ever wanted.
Verified: all 30 patches apply in order against pristine v2.13.0, and a
local libtorch[core,glog]:x64-linux build links libc10 with C10_USE_GLOG
defined -- google::IsGoogleLoggingInitialized() resolves and no
glog_internal_namespace_ reference to it remains.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pytorch 2.13 configure_file()s torch/headeronly/version.h into the source tree from caffe2/CMakeLists.txt, after torch/headeronly/CMakeLists.txt has already taken its CONFIGURE_DEPENDS glob snapshot of *.h. ninja/MSBuild therefore re-run cmake at the start of every build, and that second configure fails for CUDA builds with 'Found two conflicting CUDA versions'. Backport the upstream fix: drop *.h from the glob and list version.h explicitly. Also drop the explanatory comment from fix-system-xnnpack.patch. Its rationale: vcpkg's xnnpack port is static only (vcpkg_check_linkage ONLY_STATIC_LIBRARY), so declaring the imported targets SHARED leaves IMPORTED_IMPLIB unset and fails CMake's generate step on Windows dynamic-linkage triplets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2c0fda5 to
4018fbf
Compare
Steve Brain (EvilMcStevil)
changed the title
Billy O'Neal (BillyONeal)
left a comment
Billy O'Neal (BillyONeal)
left a comment
There was a problem hiding this comment.
I hate to be all "vibe coding" about this but this time GPT 5.6 Sol had some more detailed comments and I don't know enough about torch to conclusively check some of these. Probably just the result of this port being a huge mess. The "quoted" bits in this review were written by it.
report.md was its full output but I tried to reproduce the important bits. (In particular I do not really agree with the "moodycamel" parts of its report as that seems to be a polyfill that is unnecessary as shown by the fact that our lab accepts the Linux build)
The only bit I'm truly "request changes" over is the XNNPACK bit. The others I can't prove or disprove enough to truly block over but since you seem to be a real user if this you might want to look.
There was a problem hiding this comment.
PyTorch 2.13 changed
USE_XNNPACKfromONtoOFFby default inCMakeLists.txt. The port still stagesxnnpack, but only sets the option for the existing x86-Windows disablement inportfile.cmake. Consequently, supported configurations silently lose the backend while retaining its dependency.
I found that this was done in pytorch/pytorch@61cead4 Based on the description in that commit maybe we should just always turn this off now?
| if(VCPKG_TARGET_ARCHITECTURE STREQUAL "x64") | ||
| list(APPEND FEATURE_OPTIONS -DUSE_FBGEMM=ON) | ||
| else() | ||
| list(APPEND FEATURE_OPTIONS -DUSE_FBGEMM=OFF) | ||
| endif() |
There was a problem hiding this comment.
The port requests FBGEMM for x64 Windows in
portfile.cmake, but upstream's processor check accepts only the literal stringsx86_64andaarch64. CMake reportsAMD64for this x64-Windows build, so the condition warns and forcibly setsUSE_FBGEMM OFFwhile the package still stages FBGEMM binaries.
There was a problem hiding this comment.
It isn't clear why there needs to be a DLL there for "linking deps" but preexisting so no change required.
There was a problem hiding this comment.
yeah this is a python issue, and it exists only so Python's _load_global_deps() can dlopen it with RTLD_GLOBAL
given python is off, we don't need it at all, but vcpkg still complains about the empty export which we silence
There was a problem hiding this comment.
If we don't need it could we just delete it?
There was a problem hiding this comment.
Bundled miniz is compiled into
torch_cpu, butportfile.cmakeinstalls only PyTorch's top-level license.
Disable XNNPACK unconditionally. PyTorch 2.13 flipped the USE_XNNPACK default to OFF (pytorch/pytorch@61cead47b8) because the backend only serves the deprecated PyTorch Mobile, with ExecuTorch as the replacement. Rather than restore the old default, follow upstream and drop the now unused xnnpack dependency. Upstream's system-XNNPACK fallback in cmake/Dependencies.cmake is gated on USE_SYSTEM_XNNPACK but not on USE_XNNPACK, and USE_SYSTEM_LIBS=ON force-sets the former with a plain set(), so it cannot be turned off from the command line. Left alone it would find_library() XNNPACK behind the disabled flag and FATAL_ERROR once the dependency is gone; that same branch is what the removed x86-windows workaround was papering over. fix-disable-xnnpack.patch adds the missing USE_XNNPACK guard, which makes the whole section inert and lets fix-system-xnnpack.patch go away. Use the vcpkg concurrentqueue port instead of the vendored third_party/concurrentqueue. Its headers install to include/concurrentqueue/moodycamel/, exactly the paths that the public c10/util/Semaphore.h includes on libstdc++, so the GCC fallback is now resolvable from the installed tree. The include_directories() call is kept because torch_cpu compiles torch/nativert sources that include Semaphore.h without linking moodycamel in non-[dist] builds. Install the bundled miniz LICENSE. The vcpkg miniz port cannot be used here: PyTorch's third_party/miniz-3.0.2 is a fork adding a public MZ_ZIP_FLAG_DO_NOT_COMPUTE_CRC32 flag (used in inline_container.cc) and an m_pSeek hook, neither of which exists upstream. It stays vendored and compiled into torch_cpu, so it needs its own notice. Teach upstream's FBGEMM architecture guard the MSVC CMAKE_SYSTEM_PROCESSOR spellings. It accepted only x86_64/aarch64, so x64-windows reported AMD64, warned, and forced USE_FBGEMM OFF while the port still staged fbgemm. The guard's own comment says it exists to reject 32-bit x86, so accepting AMD64/ARM64 matches its intent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No worries, It's a big port with lots of moving dependencies, of which I personally use very few, but I don't want to break anyone else's use case, (and I find vibe coding these updates is so much more thorough than having to manually dig into all the codebases to make informed decisions, especially as each compile take hours) |
The old one-liner said the DLL is "just for linking deps", which reads as if a consumer links it; nothing does. It is built from a zero-byte translation unit and exists purely so Python's _load_global_deps() can dlopen it with RTLD_GLOBAL, pulling its own link-time MPI/MKL/CUDA dependencies into the global symbol namespace -- see upstream Note [Global dependencies]. That matters because libtorch is otherwise loaded in a local scope, which breaks libraries like OpenMPI that dlopen plugins expecting their symbols to be globally visible. None of that applies to this port: BUILD_PYTHON=OFF, and _load_global_deps() returns immediately on Windows since there is no RTLD_GLOBAL there. It ships anyway because upstream guards the target only on BUILD_SHARED_LIBS and installs it unconditionally. Being empty, it exports nothing, which is what VCPKG_POLICY_DLLS_WITHOUT_EXPORTS is suppressing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix-system-cpuinfo.patch does add_library(clog ALIAS cpuinfo::clog) unconditionally, but the port never declared the feature that provides that target. It worked only because the xnnpack dependency pulled in cpuinfo[clog] transitively, and dropping xnnpack removed that path: _add_library cannot create ALIAS target "clog" because target "cpuinfo::clog" does not already exist. clog stopped being unconditional in the cpuinfo 2026-04-13 update, so plain cpuinfo now exports no cpuinfo::clog at all. libtorch consumes the target directly and should declare it rather than free-ride on another port's dependency. Only fxdiv was otherwise reachable through xnnpack alone, and fix-system-fxdiv.patch already wraps both of its uses in if(FALSE). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2 participants
./vcpkg x-add-version --alland committing the result.